Large-Scale Language Model Data Selection for Rare-Word Speech Recognition

ABSTRACT

A method of training a language model for rare-word speech recognition includes obtaining a set of training text samples, and obtaining a set of training utterances used for training a speech recognition model. Each training utterance in the plurality of training utterances includes audio data corresponding to an utterance and a corresponding transcription of the utterance. The method also includes applying rare word filtering on the set of training text samples to identify a subset of rare-word training text samples that include words that do not appear in the transcriptions from the set of training utterances or appear in the transcriptions from the set of training utterances less than a threshold number of times. The method further includes training the external language model on the transcriptions from the set of training utterances and the identified subset of rare-word training text samples.

CROSS REFERENCE TO RELATED APPLICATIONS

This U.S. patent application claims priority under 35 U.S.C. § 119(e) toU.S. Provisional Application 63/261,946, filed on Sep. 30, 2021. Thedisclosure of this prior application is considered part of thedisclosure of this application and is hereby incorporated by referencein its entirety.

TECHNICAL FIELD

This disclosure relates to large-scale language model data selection forrare-word speech recognition.

BACKGROUND

Automated speech recognition (ASR) systems have evolved from multiplemodels where each model had a dedicated purpose to integrated modelswhere a single neural network is used to directly map an audio waveform(i.e., input sequence) to an output sentence (i.e., output sequence).This integration has resulted in a sequence-to-sequence approach, whichgenerates a sequence of words (or graphemes) when given a sequence ofaudio features. With an integrated structure, all components of a modelmay be trained jointly as a single end-to-end (E2E) neural network.Here, an E2E model refers to a model whose architecture is constructedentirely of a neural network. A fully neural network functions withoutexternal and/or manually designed components (e.g., finite statetransducers, a lexicon, or text normalization modules). Additionally,when training E2E models, these models generally do not requirebootstrapping from decision trees or time alignments from a separatesystem. These E2E automatic speech recognition (ASR) systems have madetremendous progress, surpassing conventional ASR systems in severalcommon benchmarks including word error rates (WER). The architecture ofE2E ASR models are largely application dependent. For instance, a numberof applications that involve user interaction, such as voice-search oron-device dictation, require the model to perform recognition in astreaming fashion. Other applications, like offline video captioning, donot require the model to be streaming and can make use of future contextto improve performance. Additionally, existing E2E models experiencehigh failure rates in recognizing rare words not seen during training.Rare word recognition is improved by training an external language modelon large-scale training datasets.

SUMMARY

One aspect of the disclosure provides a computer-implemented method oftraining a language model for rare-word speech recognition. Thecomputer-implemented method when executed on data processing hardwarecauses the data processing hardware to perform operations that includeobtaining a set of training text samples, and obtaining a set oftraining utterances used for training an automatic speech recognition(ASR) model. Each training utterance in the plurality of trainingutterances includes audio data corresponding to an utterance and acorresponding transcription of the utterance. The operations alsoinclude applying rare word filtering on the set of training text samplesto identify a subset of rare-word training text samples that includewords that do not appear in the transcriptions from the set of trainingutterances or appear in the transcriptions from the set of trainingutterances less than a threshold number of times. The operations furtherinclude training the external language model on the transcriptions fromthe set of training utterances and the identified subset of rare-wordtraining text samples.

Implementations of the disclosure may include one or more of thefollowing optional features. In some implementations, obtaining the setof training text samples includes receiving a corpus of training textsamples, executing a resampling function on the corpus of training textsamples to identify high frequency text samples that occur in the corpusof training text samples, and obtaining the set of training text samplesby removing the identified high frequency text samples from the corpusof training text samples. In some examples, the resampling functionincludes one of a simple power resampling function, a forced powerresampling function, or a soft logarithmic resampling function.

In some implementations, the operations further include applyingcontrastive filtering on the set of training text samples to identify asubset of target domain training text samples that match a target domainassociated with the set of training utterances. Here, training theexternal language model on the transcriptions from the set of trainingutterances and the identified subset of rare-word training text samplesfurther includes training the external language model on the identifiedsubset of target domain training text samples that match the targetdomain. In some examples, the external language model includes anexternal neural language model. In these examples, the external neurallanguage model may include a stack of conformer layers or transformerlayers.

In some implementations, the operations further include integrating thetrained external language model with the trained ASR model. The trainedexternal language model is configured to rescore probabilitydistributions over possible speech recognition hypotheses predicted bythe trained ASR model. In these implementations, the ASR model includesa first encoder, a second encoder, and a decoder. The first encoder isconfigured to receive, as input, a sequence of acoustic frames, andgenerate, at each of a plurality of output steps, a first higher orderfeature representation for a corresponding acoustic frame in thesequence of acoustic frames. The second encoder is configured toreceive, as input, the first higher order feature representationgenerated by the first encoder at each of the plurality of output steps,and generate, at each of the plurality of output steps, a second higherorder feature representation for a corresponding first higher orderfeature frame. The decoder is configured to receive, as input, thesecond higher order feature representation generated by the secondencoder at each of the plurality of output steps, and generate, at eachof the plurality of time steps, a first probability distribution overpossible speech recognition hypotheses.

In these implementations, the decoder may be further configured toreceive, as input, the first higher order feature representationgenerated by the first encoder at each of the plurality of output steps,and generate, at each of the plurality of time steps, a secondprobability distribution over possible speech recognition hypothesis.Additionally, the decoder may include a prediction network and a jointnetwork. When the ASR model is operating in a streaming mode, theprediction network is configured to receive, as input, the averageembedding generated by the prediction network at each of the pluralityof output steps and the first higher order feature representationgenerated by the first encoder at each of the plurality of output steps,and generate, at each of the plurality of output steps, the secondprobability distribution over possible speech recognition hypothesis.Alternatively, when the ASR model is operating in a non-streaming mode,the prediction network is configured to receive, as input, the averageembedding generated by the prediction network at each of the pluralityof output steps and the second higher order feature representationgenerated by the second encoder at each of the plurality of outputsteps, and generate the first probability distribution over possiblespeech recognition hypothesis.

Additionally or alternatively, the first encoder may include a causalencoder including an initial stack of conformer layers. Here, the secondencoder may include a non-causal encoder including a final stack ofconformer layers overlain on the initial stack of conformer layers. Thefirst encoder and the second encoder of the ASR model may be trainedusing Hybrid Autoregressive Transducer Factorization to facilitate theintegration of the external language model trained on text-only dataincluding the transcriptions from the set of training utterances and theidentified subset of rare-word training text samples.

Another aspect of the disclosure provides a system for training alanguage model for rare-word speech recognition. The system includesdata processing hardware and memory hardware in communication with thedata processing hardware. The memory hardware stores instructions thatwhen executed on the data processing hardware causes the date processinghardware to perform operations including obtaining a set of trainingtext samples, and obtaining a set of training utterances used fortraining an automatic speech recognition (ASR) model. Each trainingutterance in the plurality of training utterances includes audio datacorresponding to an utterance and a corresponding transcription of theutterance. The operations also include applying rare word filtering onthe set of training text samples to identify a subset of rare-wordtraining text samples that include words that do not appear in thetranscriptions from the set of training utterances or appear in thetranscriptions from the set of training utterances less than a thresholdnumber of times. The operations further include training the externallanguage model on the transcriptions from the set of training utterancesand the identified subset of rare-word training text samples.

This aspect may include one or more of the following optional features.In some implementations, obtaining the set of training text samplesincludes receiving a corpus of training text samples, executing aresampling function on the corpus of training text samples to identifyhigh frequency text samples that occur in the corpus of training textsamples, and obtaining the set of training text samples by removing theidentified high frequency text samples from the corpus of training textsamples. In some examples, the resampling function includes one of asimple power resampling function, a forced power resampling function, ora soft logarithmic resampling function.

In some implementations, the operations further include applyingcontrastive filtering on the set of training text samples to identify asubset of target domain training text samples that match a target domainassociated with the set of training utterances. Here, training theexternal language model on the transcriptions from the set of trainingutterances and the identified subset of rare-word training text samplesfurther includes training the external language model on the identifiedsubset of target domain training text samples that match the targetdomain. In some examples, the external language model includes anexternal neural language model. In these examples, the external neurallanguage model may include a stack of conformer layers or transformerlayers.

In some implementations, the operations further include integrating thetrained external language model with the trained ASR model. The trainedexternal language model is configured to rescore probabilitydistributions over possible speech recognition hypotheses predicted bythe trained ASR model. In these implementations, the ASR model includesa first encoder, a second encoder, and a decoder. The first encoder isconfigured to receive, as input, a sequence of acoustic frames, andgenerate, at each of a plurality of output steps, a first higher orderfeature representation for a corresponding acoustic frame in thesequence of acoustic frames. The second encoder is configured toreceive, as input, the first higher order feature representationgenerated by the first encoder at each of the plurality of output steps,and generate, at each of the plurality of output steps, a second higherorder feature representation for a corresponding first higher orderfeature frame. The decoder is configured to receive, as input, thesecond higher order feature representation generated by the secondencoder at each of the plurality of output steps, and generate, at eachof the plurality of time steps, a first probability distribution overpossible speech recognition hypotheses.

In these implementations, the decoder may be further configured toreceive, as input, the first higher order feature representationgenerated by the first encoder at each of the plurality of output steps,and generate, at each of the plurality of time steps, a secondprobability distribution over possible speech recognition hypothesis.Additionally, the decoder may include a prediction network and a jointnetwork. When the ASR model is operating in a streaming mode, theprediction network is configured to receive, as input, the averageembedding generated by the prediction network at each of the pluralityof output steps and the first higher order feature representationgenerated by the first encoder at each of the plurality of output steps,and generate, at each of the plurality of output steps, the secondprobability distribution over possible speech recognition hypothesis.Alternatively, when the ASR model is operating in a non-streaming mode,the prediction network is configured to receive, as input the averageembedding generated by the prediction network at each of the pluralityof output steps and the second higher order feature representationgenerated by the second encoder at each of the plurality of outputsteps, and generate the first probability distribution over possiblespeech recognition hypothesis.

Additionally or alternatively, the first encoder may include a causalencoder including an initial stack of conformer layers. Here, the secondencoder may include a non-causal encoder including a final stack ofconformer layers overlain on the initial stack of conformer layers. Thefirst encoder and the second encoder of the speech recognition model maybe trained using Hybrid Autoregressive Transducer Factorization tofacilitate the integration of the external language model trained ontext-only data including the transcriptions from the set of trainingutterances and the identified subset of rare-word training text samples.

The details of one or more implementations of the disclosure are setforth in the accompanying drawings and the description below. Otheraspects, features, and advantages will be apparent from the descriptionand drawings, and from the claims.

DESCRIPTION OF DRAWINGS

FIGS. 1A and 1B are schematic views of example speech environments usinga speech recognition model and external language model architecture forautomatic speech recognition.

FIG. 2 is a schematic view of the speech recognition model and thelanguage model architecture of FIG. 1 .

FIG. 3 is a schematic view of an example tied and reduced predictionlayer of a prediction network of the speech recognition model of FIG. 2.

FIG. 4 is a schematic view of a data selection pipeline for training alanguage model.

FIG. 5 is a schematic view of an example arrangement of operations for amethod of training a language model.

FIG. 6 is a schematic view of an example computing device that may beused to implement the systems and methods described herein.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

End-to-end (E2E) automatic speech recognition (ASR) models aretraditionally structured to operate in either a streaming mode or anon-streaming mode. Conventionally, an E2E ASR model includes an encoderand a decoder as the main components. Applications that involve end-userinteraction, like voice-search or on-device dictation, may require themodel to perform recognition in a streaming fashion, where the words areexpected to be output as they are spoken with as little latency aspossible. This prevents the use of models that use future context toimprove accuracy, such as bi-directional LSTMs. By contract,applications such as offline video captioning do not require streamingrecognition and may make full use of any available future context toimprove performance. Furthermore, conventional E2E ASR models aretrained on a small fraction of audio-text pairs as compared to over 100billion text utterances that a conventional model is trained with, andthus performs poorly on long-tail proper nouns and rare words.

Implementations herein are directed toward a single E2E ASR model incombination with an on-device neural language model trained on dataselected to improve the ASR model's recognition quality of rare words.More particularly, implementations herein are directed toward a dataselection pipeline for selecting a sufficient subset of training datasuitable for training the language model to improve recognition qualityof rare words and long-tail proper nouns. The ASR model may use cascadedencoders that include streaming and non-streaming encoders, and a singledecoder that learns to decode either using the output of the streamingor the non-streaming encoder to enable the ASR model to operate instreaming or non-streaming modes. In addition to ASR models, thearchitecture can apply to other models such as machine translation thatimplement both streaming and non-streaming modes.

FIGS. 1A and 1B are examples of a speech environment 100, 100 a-b. Inthe speech environment 100, a user's 104 manner of interacting with acomputing device, such as a user device 10, may be through voice input.The user device 10 (also referred to generally as a device 10) isconfigured to capture sounds (e.g., streaming audio data) from one ormore users 104 within the speech environment 100. Here, the streamingaudio data may refer to a spoken utterance 106 by the user 104 thatfunctions as an audible query, a command for the device 10, or anaudible communication captured by the device 10. Speech-enabled systemsof the device 10 may field the query or the command by answering thequery and/or causing the command to be performed/fulfilled by one ormore downstream applications.

The user device 10 may correspond to any computing device associatedwith a user 104 and capable of receiving audio data. Some examples ofuser devices 10 include, but are not limited to, mobile devices (e.g.,mobile phones, tablets, laptops, etc.), computers, wearable devices(e.g., smart watches), smart appliances, internet of things (IoT)devices, vehicle infotainment systems, smart displays, smart speakers,etc. The user device 10 includes data processing hardware 12 and memoryhardware 14 in communication with the data processing hardware 12 andstores instructions, that when executed by the data processing hardware12, cause the data processing hardware 12 to perform one or moreoperations. The user device 10 further includes an audio system 16 withan audio capture device (e.g., microphone) 16, 16 a for capturing andconverting spoken utterances 106 within the speech environment 100 intoelectrical signals and a speech output device (e.g., a speaker) 16, 16 bfor communicating an audible audio signal (e.g., as output audio datafrom the device 10). While the user device 10 implements a single audiocapture device 16 a in the example shown, the user device 10 mayimplement an array of audio capture devices 16 a without departing fromthe scope of the present disclosure, whereby one or more capture devices16 a in the array may not physically reside on the user device 10, butbe in communication with the audio system 16.

In the speech environment 100, an automated speech recognition (ASR)system 109 implementing an ASR model 200 (also referred to as the model200) integrated with an external language model (LM) 206 resides on theuser device 10 of the user 104 and/or on a remote computing device 60(e.g., one or more remote servers of a distributed system executing in acloud-computing environment) in communication with the user device 10via a network 40. The remote computing device 60 may include remoteresources, such as remote data processing hardware 62 (e.g., remoteservers or CPUs) and/or remote memory hardware 64 (e.g., remotedatabases or other storage hardware). The user device 10 and/or theremote computing device 60 also includes an audio subsystem 108configured to receive the utterance 106 spoken by the user 104 andcaptured by the audio capture device 16 a, and to convert the utterance106 into a corresponding digital format associated with input acousticframes 110 capable of being processed by the ASR system 109. In theexample shown in FIG. 1A, the user 104 speaks a respective utterance 106and the audio subsystem 108 converts the utterance 106 intocorresponding audio data (e.g., acoustic frames) 110 for input to theASR system 109. Thereafter, the model 200 receives, as input, the audiodata 110 corresponding to the utterance 106, and generates/predicts, asoutput, a corresponding transcription 120 (also referred to as arecognition result/hypothesis 120) of the utterance 106.

The model 200 also includes a decoder 204 (FIG. 2 ) (also referred to asa shared decoder 204) shared between its encoders which enables themodel 200 to be a single model that can operate in streaming andnon-streaming mode (e.g., in contrast with two separate models whereeach model is dedicated to either a streaming mode or non-streamingmode). For instance, as shown in FIG. 1A, a digital assistantapplication 50 executing on the user device 10 may require the speechrecognition to be streaming such that words, word pieces, and/orindividual characters appear on the screen as soon as they are spoken.Additionally, it is also likely that the user 104 of the user device 10has a low tolerance for latency when issuing queries for the digitalassistant application 50 to perform. In these scenarios where theapplication demands minimal latency, the model 200 operates in astreaming mode where the model 200 may provide streaming transcriptioncapabilities in real-time as the user 104 is speaking the utterance 106.On the other hand, when the user 104 has a higher tolerance for speechrecognition latency and/or the utterance 106 to be recognized isassociated with long-form speech (i.e., referring to speech consistingof full paragraphs or multiple sentences), the same model 200 mayoperate in a non-streaming mode and may leverage a prediction network toprovide an accurate transcription 120, but incur increased latency.

Additionally, the user 104 requires that the ASR system 109 of the userdevice 10 is able to accurately identify rare words or long-tail propernouns, which can be achieved through use of the LM 206 with the model200 to help bias the output of the model 200 when detecting rare wordsor proper nouns. As described in greater detail below with reference toFIG. 4 , the LM 206 may be trained with data sets obtained throughdiffering data selection strategies to reduce the amount of text-only 1training data needed to train the LM 206 to accurately bias the outputof the model 200 to detect rare words or proper nouns. Accordingly, theASR system 109 may implement a single ASR model that includes cascadedencoders 210, 220, for a multitude of different speech recognition tasksto provide both streaming and non-streaming transcription capabilitieswithout having to leverage separately trained ASR models on atask-by-task basis while also using the LM 206 to increase the accuracyof the transcription 120 when the utterance 106 includes rare words orlong-tail proper nouns.

In some implementations, the model 200 performs streaming encoding onthe audio data 110 first and then performs non-streaming encoding on theoutput of the streaming encoder. For instance, in the example shown, themodel 200 performs streaming speech recognition on the audio data 110using a first encoder (i.e., a low latency encoder) to produce partialspeech recognition results 120, 120 a, and non-streaming speechrecognition on the encoded audio data 110 using a second encoder (i.e.,a high latency encoder) to produce a final speech recognition result120, 120 b. Notably, the first encoder produces the partial speechrecognition results 120 a while the second encoder waits for the outputof the first encoder to produce the final speech recognition result 120b. Thus, the final speech recognition result 120 b for the inpututterance 106 may be delayed from the partial speech recognition results120 a for the input utterance by a duration.

The user device 10 and/or the remote computing device 60 also executes auser interface generator 107 configured to present a representation ofthe transcription 120 of the utterance 106 to the user 104 of the userdevice 10. As described in greater detail below, the user interfacegenerator 107 may display the partial speech recognition results 120 ain a streaming fashion during time 1 and subsequently display the finalspeech recognition result 120 b during time 2. In some configurations,the transcription 120 output from the ASR system 109 is processed, e.g.,by a natural language understanding (NLU) module executing on the userdevice 10 or the remote computing device 60, to execute a usercommand/query specified by the utterance 106. Additionally oralternatively, a text-to-speech system (not shown) (e.g., executing onany combination of the user device 10 or the remote computing device 60)may convert the transcription 120 into synthesized speech for audibleoutput by the user device 10 and/or another device.

In the example of FIG. 1A, the user 104 in the speech environment 100 ainteracts with a program or application 50 (e.g., the digital assistantapplication 50 a) of the user device 10 that uses the ASR system 109.For instance, FIG. 1A depicts the user 104 communicating with thedigital assistant application 50 a and the digital assistant application50 a displaying a digital assistant interface 18 on a screen of the userdevice 10 to depict a conversation between the user 10 and a digitalassistant of the digital assistant application 50 a. In this example,the user 104 asks the digital assistant application 50 a, “What year wasSerendipity released?” This question from the user 104 is a spokenutterance 106 captured by the audio capture device 16 a and processed byaudio systems 16 of the user device 10. In this example, the audiosystem 16 receives the spoken utterance 106 and converts it intoacoustic frames 110 for input to the ASR system 109.

Continuing with the example, the model 200, while receiving the acousticframes 110 corresponding to the utterance 106 as the user 104 speaks,encodes the acoustic frames 110 using a first encoder 210 (i.e., FIG. 2) and then decodes an encoded representation of the acoustic frames 110using a decoder 204 (FIG. 2 ) into the partial speech recognitionresults 120 a. During time 1, the user interface generator 107 presents,via the digital assistant interface 18, a representation of the partialspeech recognition results 120 a of the utterance 106 to the user 104 ofthe user device 10 in a streaming fashion such that words, word pieces,and/or individual characters appear on the screen as soon as they arespoken.

After all (or some amount) of the acoustic frames 110 corresponding tothe utterance 106 are received, and the first encoder 210 has encodedthese acoustic frames 110, the second encoder 220 (i.e., FIG. 2A)encodes the encoding output from the first encoder 210 to generate anencoding for the set of acoustic frames 110 corresponding to theutterance 106 already encoded by the first encoder 210. The decoder 204then decodes the acoustic frames 110 that have been encoded by thesecond encoder 220 and processes the decoded acoustic frames 110 usingthe LM 206, which rescores the decoded acoustic frames and generates afinal speech recognition result 120 b. For example, when the firstencoder 210 encodes all of the acoustic frames 110 corresponding to theutterance 106 (e.g., as the acoustic frames 110 are received), thesecond encoder 220 encodes all of the acoustic frames 110 that have beenencoded by the first encoder 210. In this respect, by encoding overmultiple encoded acoustic frames 110, the second encoder 210 is able toprovide greater contextual awareness (e.g., by receiving representationsof all of the acoustic frames 110 for the utterance 106) in anon-streaming fashion which may potentially reconcile or correctaspect(s) of the utterance 106 missed or misinterpreted by the streamingnature of the first encoder 210.

In some examples, an indication, such as an endpoint, that identifiesthat the user 104 has finished speaking the utterance 106 functions totrigger the second encoder 220 of the model 200 to encode all theacoustic frames 110. In other examples, the second encoder 220 encodesthe acoustic frames 110 in parallel with the first encoder 210 and thefirst encoder 210 identifies the endpoint at the end of the utterance106, thereby triggering the second encoder 220 to emit the final speechrecognition result 120 b. The endpoint identified by the first encoder210 may simultaneously trigger a microphone closing event. During time2, the user interface generator 107 presents, via the digital assistantinterface 18, a representation of the final speech recognition result120 b of the utterance 106 to the user 104 of the user device 10. Insome implementations, the user interface generator 107 replaces (ormodifies) the representation of the partial speech recognition results120 a with the representation of the final speech recognition result 120b. In this example, the utterance 106 of the user 104 contains a rareword “Serendipity” that the model 200 has not been trained on.Accordingly partial speech recognition results 120 a output by the model200 and displayed on the screen at time 1 incorrectly predicts that theutterance 106 of the user 104 is “What year was serene released?” Thefinal speech recognition result 120 b output by the model 200 anddisplayed on the screen at time 2 at increased latency improves thespeech recognition quality in terms of accuracy by identifying that theuser 104 said “Serendipity.” However, since the user interface generator107 displays the partial speech recognition results as the user speaksthe utterance 106, the higher latency associated with producing, andultimately displaying the final speech recognition result 120 b is lessnoticeable to the user 104.

In some implementations, the model 200 utilizes a pre-fetching techniquethat reduces latency by fetching speech recognition results before thefinal speech recognition result 120 b is available. Here, if the partialspeech recognition results 120 a match the final speech recognitionresult 120 b, the response fetched for the partial speech recognitionresults 120 a can be emitted instantly to save execution latency thattypically occurs after the final speech recognition result 120 b iscomplete.

In the example shown in FIG. 1A, the digital assistant application 50 amay respond to the question posed by the user 104 using natural languageprocessing. Natural language processing generally refers to a process ofinterpreting written language (e.g., the partial speech recognitionresults 120 a and/or the final speech recognition result 120 b) anddetermining whether the written language prompts any action. In thisexample, the digital assistant application 50 a uses natural languageprocessing to recognize that the question from the user 10 regards theuser's environment and more particularly a song playing in the user'svicinity. By recognizing these details with natural language processing,the automated assistant returns a response 19 to the user's query wherethe response 19 states, “Serendipity was released in 2001.” In someconfigurations, natural language processing occurs on the remotecomputing device 60 in communication with the data processing hardware12 of the user device 10.

FIG. 1B is another example of speech recognition with the ASR system 109of the speech environment 100 b. As shown in the example, the user 104interacts with a voicemail application 50, 50 b displaying a voicemailapplication interface 18, 18 b on the screen of the user device 10 totranscribe a voicemail that was left for the user 104 by Jane Doe. Inthis example, latency is not important; however, accuracy of thetranscription when processing long-tail proper nouns or rare words isimportant. The model 200 of the ASR system 109 and the LM 206 are ableto take advantage of the full context of the audio by waiting until allof the acoustic frames 110 corresponding to the voicemail are generated.This voicemail scenario also illustrates how the model 200 is capable ofhandling a long-form of speech because a voicemail is often multiplesentences or even several paragraphs. The ability to handle long-formspeech is particularly advantageous over other ASR models, such astwo-pass models with LAS decoders, because these two pass-models oftensuffer from long-form issues (e.g., a higher word deletion rate onlong-form speech) when applied to long-form conditions. For instance, byusing an RNN-T decoder as the decoder 204 in combination with cascadingencoders 202 (e.g., the first encoder 210 and the second encoder 220),the model 200 operates for both long-form speech and short-form speechwithout the long-form setbacks.

With continued reference to FIG. 1B, as discussed with respect to FIG.1A, the model 200 encodes the acoustic frames 110 using the firstencoder 210 while receiving the acoustic frames 110. After the model 200receives all of the acoustic frames 110 and encodes them with the firstencoder 210, the model 200 provides the first encoder output as input tothe second encoder 220. The second encoder 220 encodes the first encoderoutput before the decoder 204 generates an embedding and the LM 206rescores the decoder 204 output to generate the final speech recognitionresult 120 b. During time 3, the user interface generator 107 presents,via the digital assistant interface 18 b, a representation of the finalspeech recognition result 120 b without first displaying the partialspeech recognition results 120 a. For example, the final speechrecognition result 120 b is a transcript of the long-form voicemail fromJane Doe that states, “Do you want to watch Serendipity tonight? Give mea call back when you get this.”

FIG. 2 includes an example model 200 capable of operating in variouscombinations of streaming and non-streaming modes. Specifically, themodel 200 includes a cascading encoder 202, a decoder 204, and anexternal LM 206. The cascading encoder 202 refers to a model structurewhere the encoding pathway includes two encoders 210, 220 that cascadesuch that the output of one encoder 210 feeds the input of the otherencoder 220 prior to decoding. Here, the encoders 210, 220 can becascaded irrespective of the underlying architecture for each encoder.In some examples, the encoders 210, 220 include a stack of 512-dimensionconformer layers. Causal convolution and left-context attention layersmay be used for each conformer layer to strictly restrict the model useno future inputs. A multi-headed (e.g., 8 heads) attention mechanismsmay be used in a self-attention layer. The cascades encoders 210, 220may include 17 conformer layers. Here, the causal encoder 210 mayinclude 15 conformer layers while the non-causal encoder 210 may includetwo conformer layers that take in additional right context (e.g., 5.04seconds). Optionally, transformer layers may be used in lieu ofconformer layers.

In other implementations, one encoder is constructed with an LSTMstructure while the other encoder is constructed using bi-directionalLSTM layers or conformer layers (e.g., a conformer-transducer). In otherwords, the encoders 210, 220 may have different architectures or similararchitectures. For instance, the cascading encoder 202 may be roughlyanalogous to an acoustic model (AM) in a traditional ASR system, and mayinclude a recurrent network of stacked Long Short-Term Memory (LSTM)layers. Here, the first encoder 210 is a streaming encoder that includesunidirectional Long Short Term Memory (LSTM) layers while the secondencoder 220 is a non-streaming encoder that includes bidirectional LSTMlayers or conformer layers. In a cascading encoder 202, where bothencoders 210, 230 include LSTM layers, the second encoder 220 thatreceives the output of the first encoder 210 may take advantage of theLSTM layers of the first encoder 210 such that the second encoder 220includes fewer LSTM layers than the first encoder 210 (and fewer LSTMlayers than a fully non-streaming model). By having fewer LSTM layers,the cascading encoder 202 may reduce the number of more computationallyexpensive bidirectional layers, making the model 200 more streamlinedthan simply combining a traditional streaming model with a traditionalnon-streaming model. In some implementations, in order to limit theamount of future context that the cascaded encoders model 200 sees, thesecond encoder 220 uses some number of conformer layers (e.g., twolayers) with a particular amount of right context (e.g., five seconds ofright context), while the first encoder 210 continues to use LSTMlayers. For these implementations, each conformer layer in the secondencoder 220 may have 640 units to match the LSTM layers and adds around10 million additional parameters.

Still referring to FIG. 2 , the first encoder 210 reads a sequence ofd-dimensional feature vectors (e.g., acoustic frames 110 shown in FIGS.1A and 1B) x=(x₁, x₂, . . . , x_(T)), where x_(t)∈

^(d) and produces, at each time step, a first higher-order featurerepresentation. This first higher-order feature representation isdenoted as e^(s). Similarly, the second encoder 220 is connected incascade to the first encoder 210, and is trained to receive the firsthigher order feature e^(s) as input, and output a second higher orderfeature representation. This second higher order feature representationis denoted as e^(a). Both the first encoder 210 and the second encoder220 are directly connected to, and shared by, the decoder 204.Accordingly, the decoder 204 receives both the first higher orderfeature representation e^(s) and the second higher order featurerepresentation e^(a) as inputs.

The decoder 204 may include a recurrent neural network-transducer(RNN-T) architecture having a joint layer 230 and a prediction network300. The decoder 204 uses the joint layer 230 to combine (i.e., when themodel 200 operates in non-streaming mode) the first and second higherorder feature representations e^(s), e^(a), output by the cascadingencoder 202, as well as an embedding output from the embedding lookup300 for the previous prediction y_(r-1)), in order to produce a decoderoutput. The decoder output is then passed to the external LM 206 thatrescores/improves the initial outputs from the decoder 204 withtechniques such as lattice rescoring or n-best re-ranking. In otherwords, the decoder 204 produces predictions and the external LM 206finalizes the prediction by improving recognition accuracy on rare wordsor long-tail proper nouns. When the model 200 operates in the streamingmode, the joint layer 230 receives the output of the embedding lookup300 and only the first higher order feature representation e^(s) outputfrom the first encoder 210.

The decoder output can be a probability distribution, P (y_(i)|y_(i-1),. . . , y₀, X), over the current sub-word unit, y_(i), given thesequence of the N previous non-blank symbols 301 previous units,{y_(i-1), . . . , y_(i-N)}, and input, x. Although not illustrated, themodel 200 may include a Softmax layer that receives the output of thedecoder 204. In some implementations, the Softmax layer is separate fromthe decoder 204 and processes the output, y_(r), from the decoder 204.The output of the Softmax layer is then used in a beam search process toselect orthographic elements. In some implementations, the Softmax layeris integrated with the decoder 204, such that the output y_(r) of thedecoder 204 represents the output of the Softmax layer.

In some examples, the prediction network 300 has two 2,048-dimensionalLSTM layers, each of which is also followed by 640-dimensionalprojection layer, such that the LSTM-based embedding lookup 300 may haveabout 23.4 million parameters. When the prediction network 300 includesLSTM layers, to contribute to techniques for reducing the size of theprediction network 300 without sacrificing accuracy/performance of themodel 200, the prediction network 300 may include a stateless predictionnetwork that receives a limited-history sequence of non-blank symbolsy_(ui-n), . . . , y_(ui-1) limited to the N previous non-blank symbols301 output by the final Softmax layer. For instance, FIG. 3 shows thestateless prediction network 300 of the model 200 receiving, as input, asequence of non-blank symbols y_(ui-n), . . . , y_(ui-1) that is limitedto the N previous non-blank symbols 301 a-n output by the final Softmaxlayer. In some examples, N is equal to two. In other examples, N isequal to five, however, the disclosure is non-limiting and N may equalany integer. The sequence of non-blank symbols 301 a-n indicates initialspeech recognition results 120 a (FIG. 1 ). In some implementations, theprediction network 300 includes a multi-headed attention mechanism 302that shares a shared embedding matrix 304 across each head 302A-302H ofthe multi-headed attention mechanism. In one example, the multi-headedattention mechanism 302 includes four heads. However, any number ofheads may be employed by the multi-headed attention mechanism 302.Notably, the multi-headed attention mechanism improves performancesignificantly with minimal increase to model size. As described ingreater detail below, each head 302A-H includes its own row of positionvectors 308, and rather than incurring an increase in model size byconcatenating outputs 318A-H from all the heads, the outputs 318A-H areinstead averaged by a head average module 322.

Referring to the first head 302A of the multi-headed attention mechanism302, the head 302A generates, using the shared embedding matrix 304, acorresponding embedding 306, 306 a-n (e.g., X∈

^(N×d) ^(e) ) for each non-blank symbol 301 among the sequence ofnon-blank symbols y_(ui-n), . . . , y_(ui-1) received as input at thecorresponding time step from the plurality of time steps. Notably, sincethe shared embedding matrix 304 is shared across all heads of themulti-headed attention mechanism 302, the other heads 302B-H allgenerate the same corresponding embeddings 306 for each non-blanksymbol. The head 302A also assigns a respective position vectorPV_(Aa-An) 308, 308Aa-An (e.g., P∈

^(H×N×D) ^(e) ) to each corresponding non-blank symbol in the sequenceof non-blank symbols y_(ui-n), . . . , y_(ui-1). The respective positionvector PV 308 assigned to each non-blank symbol indicates a position inthe history of the sequence of non-blank symbols (e.g., the N previousnon-blank symbols output by the final Softmax layer). For instance, thefirst position vector PV_(Aa) is assigned to a most recent position inthe history, while the last position vector PV_(An) is assigned to alast position in the history of the N previous non-blank symbols outputby the final Softmax layer. Notably, each of the embeddings 306 mayinclude a same dimensionality (i.e., dimension size) as each of theposition vectors PV 308.

While the corresponding embedding generated by shared embedding matrix304 for each for each non-blank symbol 301 among the sequence ofnon-blank symbols 301 a-n,y_(ui-n), . . . , y_(ui-1), the same at all ofthe heads 302A-H of the multi-headed attention mechanism 302, each head302A-H defines a different set/row of position vectors 308. Forinstance, the first head 302A defines the row of position vectorsPV_(Aa-An) 308Aa-An, the second head 302B defines a different row ofposition vectors PV_(Ba-Bn) 308 _(Ba-Bn), . . . , and the H^(th) head302 H defines another different row of position vectors PV_(Ha-Hn) 308_(Ha-Hn).

For each non-blank symbol in the sequence of non-blank symbols 301 a-nreceived, the first head 302A also weights, via a weight layer 310, thecorresponding embedding 306 proportional to a similarity between thecorresponding embedding and the respective position vector PV 308assigned thereto. In some examples, the similarity omc;ides a cosinesimilarity (e.g., cosine distance). In the example shown, the weightlayer 310 outputs a sequence of weighted embeddings 312, 312Aa-An eachassociated the corresponding embedding 306 weighted proportional to therespective position vector PV 308 assigned thereto. Stated differently,the weighted embeddings 312 output by the weight layer 310 for eachembedding 306 may correspond to a dot product between the embedding 306and the respective position vector PV 308. The weighted embeddings 312may be interpreted as attending over the embeddings in proportion to howsimilar they are to the positioned associated with their respectiveposition vectors PV 308. To increase computational speed, the predictionnetwork 300 includes non-recurrent layers, and therefore, the sequenceof weighted embeddings 312Aa-An are not concatenated, but instead,averaged by a weighted average module 316 to generate, as output fromthe first head 302A, a weighted average 318A of the weighted embeddings312Aa-An represented by:

$\begin{matrix}{{{Prediction}\left( {X,P} \right)} = {\frac{1}{H*N}{\sum\limits_{h,n}{X_{n}*{\sum\limits_{e}\left( {X_{n,e}*P_{h,n,e}} \right)}}}}} & (1)\end{matrix}$

In Equation 1, h represents the index of the heads 302, n representsposition in context, and e represents the embedding dimension.Additionally, in Equation 1, H, N, and d_(e) include the sizes of thecorresponding dimensions. The position vector PV 308 does not have to betrainable and may include random values. Notably, even though theweighted embeddings 312 are averaged, the position vectors PV 308 canpotentially save position history information, alleviating the need toprovide recurrent connections at each layer of the prediction network300.

The operations described above with respect to the first head 302A, aresimilarly performed by each other head 302B-H of the multi-headedattention mechanism 302. Due to the different set of positioned vectorsPV 308 defined by each head 302, the weight layer 310 outputs a sequenceof weighted embeddings 312 _(Ba-Bn), 312 _(Ha-Hn) at each other head302B-H that is different than the sequence of weighted embeddings 312_(Aa-Aa) at the first head 302A. Thereafter, the weighted average module316 generates, as output from each other corresponding head 302B-H, arespective weighted average 318B-H of the corresponding weightedembeddings 312 of the sequence of non-blank symbols.

In the example shown, the prediction network 300 includes a head averagemodule 322 that averages the weighted averages 318A-H output from thecorresponding heads 302A-H. A projection layer 326 with SWISH mayreceive, as input, an output 324 from the head average module 322 thatcorresponds to the average of the weighted averages 318A-H, andgenerate, as output, a projected output 328. A final layer normalization330 may normalize the projected output 328 to provide the singleembedding vector Pu_(i) 350 at the corresponding time step from theplurality of time steps. The prediction network 300 generates only asingle embedding vector Pu_(i) 350 at each of the plurality of timesteps subsequent to an initial time step.

In some configurations, the prediction network 300 does not implementthe multi-headed attention mechanism 302 and only performs theoperations described above with respect to the first head 302A. In theseconfigurations, the weighted average 318A of the weighted embeddings312Aa-An is simply passed through the projection layer 326 and layernormalization 330 to provide the single embedding vector Pu_(i) 350.

In other configurations, the prediction network 300 may instead includeconformer or transformer layers in lieu of LSTM layers. In otherexamples, the prediction network 300 includes a V2 embedding look uptable in lieu of a network of LSTM, transformer, or conformer layers. Ateach time step, the V2 embedding lookup table may receive, as input, theprevious two predictions (e.g., 1-hot vectors) output by the joint layer230, compute a respective embedding d₁, d₂ for each of the previous twopredictions, and provide a concatenated output [d₁, d₂] to the jointlayer 230. Comparatively, the V2 embedding lookup table may have onlyabout two (2) million parameters, whereas an LSTM-based predictionnetwork may include about 23.4 million parameters. Finally, the jointlayer 230 may also be a one-layer neural network with 640 hidden units.The Softmax layer may be composed of a unified word piece or graphemeset that is generated using all unique word pieces or graphemes in aplurality of training data sets.

The decoder 204 is configured to generate, at each output step, aprobability distribution over possible speech recognition hypotheses.Stated differently, the joint layer 230 generates, at each output step(e.g., time step), a probability distribution over possible speechrecognition hypotheses. Here, the “possible speech recognitionhypotheses” correspond to a set of output labels/symbols (also referredto as “speech units”) each representing a grapheme (e.g.,symbol/character) or a word piece in a specified natural language. Forexample, when the natural language is English, the set of output labelsmay include twenty-seven (27) symbols, e.g., one label for each of the26-letters in the English alphabet and one label designating a space.Accordingly, the joint layer 230 may output a set of values indicativeof the likelihood of occurrence of each of a predetermined set of outputlabels. This set of values can be a vector (e.g., a one-hot vector) andcan indicate a probability distribution over the set of output labels.In some cases, the output labels are graphemes (e.g., individualcharacters, and potentially punctuation and other symbols), but the setof output labels is not so limited. For example, the set of outputlabels can include wordpieces and/or entire words, in addition to orinstead of graphemes. The output labels could also be other types ofspeech units, such as phonemes or sub-phonemes. The output distributionof the joint layer 230 can include a posterior probability value foreach of the different output labels. Thus, if there are 100 differentoutput labels representing different graphemes or other symbols, theoutput of the joint layer 230 can include 100 different probabilityvalues, one for each output label. The probability distribution can thenbe used to select and assign scores to candidate orthographic elements(e.g., graphemes, wordpieces, and/or words) in a beam search process(e.g., by the Softmax layer) for determining the transcription 120.

In some implementations, the LM 206 includes a unidirectional conformerthat looks back a predetermined number of tokens (e.g., seven tokens)for each output wordpiece model prediction. The conformer LM 206 mayhave a stack of layers (e.g., 12 layers) where each layer includes amodel dimension of 768, a feedforward layer dimension of 2048, and asix-head attention. In these implementations, the conformer LM 206 istrained to predict 4,096 wordpieces.

Integrating ASR models with external LMs typically requires shallowfusion. However, overconfidence of the cascading encoder 202 and thedecoder 204 can make weighting difficult and often lead to highdeletions of words. Accordingly, a Hybrid Autoregressive Transducer(HAT) model may be utilized to factor out an internal loss languagemodel score p_(ILM(y)) of the model 200 so that the effective score ofthe model 200 can be represented as follows.

log p(x|y)≈log p(y|x)−log plm(y)  (2)

Accordingly, HAT factorization allows the integration of the model 200with the external LM 206 without requiring coverage penalties asfollows.

y*=arg max_(y)[λ₁log p(Y|x)−λ₂ log pilm(y)+log plm(y)]  (3)

where λ₁ and λ₂ denote weights assigned to the external LM 206 and theinternal language model, respectively. By using HAT factorization duringthe training process 300, the LM 206 is better integrated with thecascading encoder 202 and decoder 204.

Continuing with the example in FIG. 2 , in some implementations, themodel 200 operates in both the streaming and non-streaming modes inparallel. When operating in both streaming and non-streaming mode at thesame time, the model 200 first performs streaming speech recognition onthe audio data 110 using the first encoder 210 to generate the firsthigher order representation e^(s) for both the second encoder 220 andthe decoder 204. The decoder 204 then produces the partial speechrecognition results 120, 120 a. The model 200 also performsnon-streaming speech recognition on the encoded audio data 110 where thesecond encoder 220 uses the first higher order representation e^(s)received from the first encoder 210 to generate the second higher orderrepresentation e^(a). The decoder 204 then produces a speech recognitionresult, which is then rescored by the LM 206 to produce the final speechrecognition result 120, 120 b. As noted by the time, the first encoder210 produces the partial speech recognition results 120 a while thesecond encoder 220 waits for the output of the first encoder 210.Finally, the LM 206 may bias the output from the decoder 204 to generatethe final speech recognition result 120 b. Thus, the final speechrecognition result 120 b for the input utterance 106 may be delayed fromthe partial speech recognition results 120 a for the input utterance. Asmentioned previously, the first encoder 210 may identify an endpoint ofthe utterance 106 that triggers a microphone closing event and triggersthe final speech recognition result 120 b to be emitted.

In some implementations, to further reduce the size of the decoder 204,i.e., the prediction network 300 and the joint layer 230, parametertying between the prediction network 300 and the joint layer 230 isapplied. Specifically, for a vocabulary size |V| and an embeddingdimension d_(e), the shared embedding matrix 304 at the predictionnetwork 300 is E∈

^(|V|×d) ^(e) Meanwhile, a last hidden layer includes a dimension sized_(h) at the joint layer 230, feed-forward projection weights from thehidden layer to the output logits will be W∈

^(d) ^(h) ^(x|V+1|), with an extra blank token in the vocabulary.Accordingly, the feed-forward layer corresponding to the last layer ofthe joint layer 230 includes a weight matrix [d_(h), |V]|. By having theprediction network 300 to tie the size of the embedding dimension d_(e)to the dimensionality d_(h) of the last hidden layer of the joint layer230, the feed-forward projection weights of the joint layer 230 and theshared embedding matrix 304 of the prediction network 300 can sharetheir weights for all non-blank symbols via a simple transposetransformation. Since the two matrices share all their values, thedecoder 204 only needs to store the values once on memory, instead ofstoring two individual matrices. By setting the size of the embeddingdimension d_(e) equal to the size of the hidden layer dimension d_(h),the decoder 204 reduces a number of parameters equal to the product ofthe embedding dimension d_(e) and the vocabulary size |V|. This weighttying corresponds to a regularization technique.

FIG. 4 shows an example of a data selection pipeline 400 for trainingthe external LM 206 of the ASR system 109. Generally, large trainingdata sets, such as a corpus of training text samples 412, 412 a-n areused to train language models that execute in computing environments(e.g., the cloud) that are not inhibited by processing, memory/storage,and power constraints. However, in some configurations, the ASR system109 resides on the user device 10 of the user 104, thereby limiting thenumber of parameters in the LM 206, and consequently, the number oftraining samples in the training set used to train the LM 206. Toresolve this, a data selection pipeline 400 processes the corpus oftraining text samples 412 to reduce the amount of training data (i.e.,training text samples) needed to train the LM 206 to accuratelyrecognize rare words, thereby allowing the ASR system 109 including theexternal LM 206 to run on-device where processing and/or memory/storageresources is limited. In other words, the data selection pipeline 400filters the training text samples from the corpus to identify a subsetof training samples sufficient for improving rare-word recognition. Thecorpus of training text samples 412 may include 213 billion sentenceswith a size of about 12 terabytes, wherein 7.2 billion of the samplesare distinct. The pipeline 400 may reduce the number of samples 412 inthe corpus to about only four (4) billion sentences for training the LM206, which is 53× smaller than the original corpus and withoutdegradation in overall performance of the LM 206 in terms of word errorrate (WER).

As shown in FIG. 4 , the data selection pipeline 400 uses a resamplingfilter 420, a rare word filter 430, and a contrastive filter 440 toreduce the number of the training text utterances in the corpus oftraining text samples 412 for training the LM 206. The data selectionpipeline 400 obtains the corpus of training text samples 412, 412 a-nstored in a training text data store 410 and a plurality of trainingutterances 452, 452 a-n stored in a training utterances data store 450.The training utterances 452 stored in the training utterances data store450 are used for training the ASR model 200 and each training utterance452 includes audio data 454 corresponding to an utterance and acorresponding transcription 456 of the utterance. While FIG. 4 shows thepipeline 400 employing each of the resampling filter 420, the rare wordfilter 430, and the contrastive filter 440 for selecting training textdata, the pipeline may use only one or two of the resampling filter 420,the rare word filter 430, and the contrastive filer 440 for selectingtraining text data.

The resampling filter 420 receives the corpus of training text samples412 stored in the training text data store 410 and executes a resamplingfunction to identify rare words (e.g., words that occur less frequently)in the corpus by identifying and removing high frequency training textsamples from the corpus to output a set of low frequency training textsamples (also referred to as ‘set of training text samples’) 422corresponding samples from the corpus of training text samples 412 thatinclude rare words. In the example shown, the resampling filter 420measures frequency at the sentence level rather than at the word levelfor the sake of simplicity. The resampling filter 420 may, however,measure rareness of a sentence from an aggregate of its own wordswithout departing from the scope of the present disclosure. As usedherein, a word or sentence is more rare when it has a lower frequency(there are fewer occurrences of it) in the corpus relative to otherwords or sentences. The term “tailedness” may be used to describe therelative amount of rare words occurring the corpus of training textsamples 412. The frequency distribution of the corpus of training textsamples 412 as a whole is linear on a log-log plot and is expressed by:

distinct_count(f)≈Af^(−∝)  (4)

where f denotes the frequency and A denotes the number of distincttraining text samples 412 (i.e., having a frequency f of one). Bychanging the power α, the distribution changes. For example, a larger αresults in a distribution with a heavy frequency of rare words. Exampleswhere α approaches infinity indicate that there are no duplicatetraining texts 412 in the plurality of training text samples 412.However, the plurality of training text samples 412 stored in thetraining text data store 410 include an α of 1.1-2.5. Furthermore,training text samples 412 occurring at an excessive frequency rate(e.g., “home” in a Maps domain) deviate from the linear distribution ofthe frequency distribution.

To filter the high frequency training texts from the corpus of trainingtext samples 412, thereby increasing the number of rare words in the setof low frequency training text samples 422, the resampling filter 420may execute a resampling function including one of a simple powerresampling function, a forced power resampling function, or a softlogarithmic resampling function. Simple power resampling may includetuning the rareness of the frequency distribution distinct_count(f) byapplying a parameter β. The simple power frequency distribution may thenbe expressed as Af^(−αβ). In other implementations, forced powerresampling is used to manage the excessive frequency training textsamples in the corpus of training text samples 412 by forcing eachtraining text 412 to fit a line fit. For example, the line fit for aMaps domain may indicate a distinct count of 1 corresponding to afrequency of 10⁶. In these examples, for each training text sample thathas a distinct_count of 1, its resampled frequency f₁ will be 10⁶regardless of its original frequency f₀. In this example, a trainingtext sample with a high original frequency f₀ (e.g., 10⁸) is forced to aresampled frequency f₁ of 10⁶. This forced power resampling operation isexpressed as:

$\begin{matrix}{{F(f)}:={❘\frac{{distint\_ count}(f)}{A}❘}^{\propto}} & (5)\end{matrix}$

Alternatively, the resampling filter 420 may execute a soft logarithmicresampling function, which matches the original frequency distributiondistinct_count(f) of the corpus of training text samples 412 and thenremoves training texts from the corpus that exceed a threshold. The softlogarithmic function is expressed by:

$\begin{matrix}{f_{1} = {f_{c}{\log\left( {1 + \frac{f_{0}}{f_{c}}} \right)}}} & (6)\end{matrix}$

Where f_(c) denotes a threshold frequency.

Once the resampling filter 420 removes the high frequency training textsfrom the corpus of training text samples 412 to output the set oftraining text samples 422 that include rare words, the set of trainingtext samples 422 are provided as input to the rare word filter 430 andthe contrastive filter 440. Notably, the removal of high frequencytraining text samples from the corpus is desirable since these sampleswould provide a distributional bias that may prevent the LM 206 fromlearning a long tail of form the corpus that includes many rare words.The rare word filter 430 identifies a subset of rare-word training textsamples 432 that include words that do not appear in the transcriptions456 from the set of training utterances 452 or appear in thetranscriptions 456 from the set of training utterances 452 less than athreshold number of times. Likewise, the contrastive filter 440identifies a subset of target domain training text samples 442 withinthe set training text samples 422 that match a target domain associatedwith the training utterances 452 used to train the ASR model 200. Thetraining utterances 452 may be referred to as ASR training utterances452 each including ASR audio data 454 paired with corresponding ASRtranscripts 456. The data selection pipeline 400 then combines ASRtranscripts 456, the rare word training text samples 432, and the targetdomain training text samples 442 into mini-batches for use by a languagemodel trainer 480 to train the LM 206. The mini-batches may be combinedaccording to a sampling ratio (e.g., 20%/40%/40% for ASR transcriptions456/rare word training text samples 432/target domain training textsamples 442).

The rare word filter 430 directly filters the transcriptions 456 fromthe set of ASR training utterances 452 that include words that appear inthe set of training text samples 422 using a frequency threshold f_(t)(e.g., 15) to identify training text samples for inclusion in the subsetof rare-word training text samples 432. The rare word filter 420 alsoidentifies any training text samples 422 that do not appear in thetranscriptions 456 for inclusion in the subset of rare-word trainingtext samples 432. The contrastive filter 440 applies contrastiveselection/filtering on the set of low frequency training text samples422 output by the resampling filter 420 to identify a subset of targetdomain training text samples 442 that match a target domain associatedwith the set of training utterances 452 used to train the ASR model 200.The corpus of training text samples 412 may include text samplescollected from domains that are different than the domain the ASR model200 is trained to recognize speech. For instance, the text samples maycollected form typed search queries containing more website names whilethe target domain of the ASR model 200 corresponds to voice searchcontaining more voice commands. This contrastive selection is calculatedfor each training text sample in the set of low frequency training textsamples 422 by:

score(x)=

_(target)(x)−

_(background)(x)  (7)

where

denotes the logarithmic perplexity of the training text sample 422,target denotes the target LM 206, and background denotes a backgroundLanguage Model trained on a fully deduplicated set of training data. Thecontrastive selection is then tuned on the transcriptions 456 of thetraining utterances 452 to produce the target LM 206. The score for atraining text sample will be lower when the training text sample iscloser to the transcriptions 456 of the training utterances 452 used totrain the ASR model 200. The contrastive filter 440 then may discard atraining text sample 422 that is above a threshold, to identify thesubset of target domain training texts 442 from the set of low frequencytraining text samples 422 that are below the threshold. As used herein,a target domain associated with the training utterances may includeassistant queries, voice search queries, navigation queries, orutterances associated with any other domain. Notably, the ASR model 200of FIGS. 1-3 is trained on the training utterances 452 that each includeaudio data 454 of a corresponding utterance and a correspondingtranscription 456 of the utterance that serves as a ground-truth labelfor the audio data 454.

FIG. 5 includes a flowchart of an example arrangement of operations fora method 500 of training a language model 206 for rare-word speechrecognition. The method 500 includes, at operation 502, obtaining a setof training text samples 422. At operation 504, the method 500 alsoincludes obtaining a set of training utterances 452 used for training aspeech recognition model 200 (e.g., an automatic speech recognition(ASR) model 200), each training utterance 452 in the plurality oftraining utterances 452 including audio data 454 corresponding to anutterance and a corresponding transcription 456 of the utterance.

At operation 506, the method 500 includes applying rare word filteringon the set of training text samples 422 to identify a subset ofrare-word training text samples 432. The subset of rare-word trainingtext samples 432 include words that do not appear in the transcriptions456 from the set of training utterances 452 or appear in thetranscriptions 456 from the set of training utterances 452 less than athreshold number of times. The method 500 further includes, at operation508, training the external language model 206 on the transcriptions 456from the set of training utterances 452 and the identified subset ofrare-word training text samples 432.

FIG. 6 is schematic view of an example computing device 600 that may beused to implement the systems and methods described in this document.The computing device 600 is intended to represent various forms ofdigital computers, such as laptops, desktops, workstations, personaldigital assistants, servers, blade servers, mainframes, and otherappropriate computers. The components shown here, their connections andrelationships, and their functions, are meant to be exemplary only, andare not meant to limit implementations of the implementations describedand/or claimed in this document.

The computing device 600 includes a processor 610, memory 620, a storagedevice 630, a high-speed interface/controller 640 connecting to thememory 620 and high-speed expansion ports 650, and a low speedinterface/controller 660 connecting to a low speed bus 670 and a storagedevice 630. Each of the components 610, 620, 630, 640, 650, and 660, areinterconnected using various busses, and may be mounted on a commonmotherboard or in other manners as appropriate. The processor 610 (alsoreferred to as “data processing hardware 610” that may include the dataprocessing hardware 12 of the user device 10 or the data processinghardware 62 of the remote computing device 60) can process instructionsfor execution within the computing device 600, including instructionsstored in the memory 620 or on the storage device 630 to displaygraphical information for a graphical user interface (GUI) on anexternal input/output device, such as display 680 coupled to high speedinterface 640. In other implementations, multiple processors and/ormultiple buses may be used, as appropriate, along with multiple memoriesand types of memory. Also, multiple computing devices 600 may beconnected, with each device providing portions of the necessaryoperations (e.g., as a server bank, a group of blade servers, or amulti-processor system).

The memory 620 (also referred to as “memory hardware 620” that mayinclude the memory hardware 14 of the user computing device 10 or thememory hardware 64 of the remote computing device 60) stores informationnon-transitorily within the computing device 600. The memory 620 may bea computer-readable medium, a volatile memory unit(s), or non-volatilememory unit(s). The non-transitory memory 620 may be physical devicesused to store programs (e.g., sequences of instructions) or data (e.g.,program state information) on a temporary or permanent basis for use bythe computing device 600. Examples of non-volatile memory include, butare not limited to, flash memory and read-only memory (ROM)/programmableread-only memory (PROM)/erasable programmable read-only memory(EPROM)/electronically erasable programmable read-only memory (EEPROM)(e.g., typically used for firmware, such as boot programs). Examples ofvolatile memory include, but are not limited to, random access memory(RAM), dynamic random access memory (DRAM), static random access memory(SRAM), phase change memory (PCM) as well as disks or tapes.

The storage device 630 is capable of providing mass storage for thecomputing device 600. In some implementations, the storage device 630 isa computer-readable medium. In various different implementations, thestorage device 630 may be a floppy disk device, a hard disk device, anoptical disk device, or a tape device, a flash memory or other similarsolid state memory device, or an array of devices, including devices ina storage area network or other configurations. In additionalimplementations, a computer program product is tangibly embodied in aninformation carrier. The computer program product contains instructionsthat, when executed, perform one or more methods, such as thosedescribed above. The information carrier is a computer- ormachine-readable medium, such as the memory 620, the storage device 630,or memory on processor 610.

The high speed controller 640 manages bandwidth-intensive operations forthe computing device 600, while the low speed controller 660 manageslower bandwidth-intensive operations. Such allocation of duties isexemplary only. In some implementations, the high-speed controller 640is coupled to the memory 620, the display 680 (e.g., through a graphicsprocessor or accelerator), and to the high-speed expansion ports 650,which may accept various expansion cards (not shown). In someimplementations, the low-speed controller 660 is coupled to the storagedevice 630 and a low-speed expansion port 690. The low-speed expansionport 690, which may include various communication ports (e.g., USB,Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or moreinput/output devices, such as a keyboard, a pointing device, a scanner,or a networking device such as a switch or router, e.g., through anetwork adapter.

The computing device 600 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as astandard server 600 a or multiple times in a group of such servers 600a, as a laptop computer 600 b, or as part of a rack server system 600 c.

Various implementations of the systems and techniques described hereincan be realized in digital electronic and/or optical circuitry,integrated circuitry, specially designed ASICs (application specificintegrated circuits), computer hardware, firmware, software, and/orcombinations thereof. These various implementations can includeimplementation in one or more computer programs that are executableand/or interpretable on a programmable system including at least oneprogrammable processor, which may be special or general purpose, coupledto receive data and instructions from, and to transmit data andinstructions to, a storage system, at least one input device, and atleast one output device.

A software application (i.e., a software resource) may refer to computersoftware that causes a computing device to perform a task. In someexamples, a software application may be referred to as an “application,”an “app,” or a “program.” Example applications include, but are notlimited to, system diagnostic applications, system managementapplications, system maintenance applications, word processingapplications, spreadsheet applications, messaging applications, mediastreaming applications, social networking applications, and gamingapplications.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms “machine-readable medium” and“computer-readable medium” refer to any computer program product,non-transitory computer readable medium, apparatus and/or device (e.g.,magnetic discs, optical disks, memory, Programmable Logic Devices(PLDs)) used to provide machine instructions and/or data to aprogrammable processor, including a machine-readable medium thatreceives machine instructions as a machine-readable signal. The term“machine-readable signal” refers to any signal used to provide machineinstructions and/or data to a programmable processor.

The processes and logic flows described in this specification can beperformed by one or more programmable processors, also referred to asdata processing hardware, executing one or more computer programs toperform functions by operating on input data and generating output. Theprocesses and logic flows can also be performed by special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit). Processors suitable for theexecution of a computer program include, by way of example, both generaland special purpose microprocessors, and any one or more processors ofany kind of digital computer. Generally, a processor will receiveinstructions and data from a read only memory or a random access memoryor both. The essential elements of a computer are a processor forperforming instructions and one or more memory devices for storinginstructions and data. Generally, a computer will also include, or beoperatively coupled to receive data from or transfer data to, or both,one or more mass storage devices for storing data, e.g., magnetic,magneto optical disks, or optical disks. However, a computer need nothave such devices. Computer readable media suitable for storing computerprogram instructions and data include all forms of non-volatile memory,media and memory devices, including by way of example semiconductormemory devices, e.g., EPROM, EEPROM, and flash memory devices; magneticdisks, e.g., internal hard disks or removable disks; magneto opticaldisks; and CD ROM and DVD-ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, one or more aspects of thedisclosure can be implemented on a computer having a display device,e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, ortouch screen for displaying information to the user and optionally akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's client device in response to requests received from the webbrowser.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made without departingfrom the spirit and scope of the disclosure. Accordingly, otherimplementations are within the scope of the following claims.

What is claimed is:
 1. A computer-implemented method for training anexternal language model to recognize rare words in speech, thecomputer-implemented method when executed on data processing hardwarecauses the data processing hardware to perform operations comprising:obtaining a set of training text samples; obtaining a set of trainingutterances used for training an automatic speech recognition (ASR)model, each training utterance in the plurality of training utterancescomprising audio data corresponding to an utterance and a correspondingtranscription of the utterance; applying rare word filtering on the setof training text samples to identify a subset of rare-word training textsamples that include words that do not appear in the transcriptions fromthe set of training utterances or appear in the transcriptions from theset of training utterances less than a threshold number of times; andtraining the external language model on the transcriptions from the setof training utterances and the identified subset of rare-word trainingtext samples.
 2. The computer-implemented method of claim 1, whereinobtaining the set of training text samples comprises: receiving a corpusof training text samples; executing a resampling function on the corpusof training text samples to identify high frequency text samples thatoccur in the corpus of training text samples; and obtaining the set oftraining text samples by removing the identified high frequency textsamples from the corpus of training text samples.
 3. Thecomputer-implemented method of claim 1, wherein the resampling functioncomprises one of a simple power resampling function, a forced powerresampling function, or a soft logarithmic resampling function.
 4. Thecomputer-implemented method of claim 1, wherein the operations furthercomprise: applying contrastive filtering on the set of training textsamples to identify a subset of target domain training text samples thatmatch a target domain associated with the set of training utterances,wherein training the external language model on the transcriptions fromthe set of training utterances and the identified subset of rare-wordtraining text samples further comprises training the external languagemodel on the identified subset of target domain training text samplesthat match the target domain.
 5. The computer-implemented method ofclaim 1, wherein the external language model comprises an externalneural language model.
 6. The computer-implemented method of claim 5,wherein the external neural language model comprises a stack ofconformer layers or transformer layers.
 7. The computer-implementedmethod of claim 1, wherein the operations further comprise integratingthe trained external language model with the trained ASR model, thetrained external language model configured to rescore probabilitydistributions over possible speech recognition hypotheses predicted bythe trained ASR model.
 8. The computer-implemented method of claim 7,wherein the ASR model comprises: a first encoder configured to: receive,as input, a sequence of acoustic frames; and generate, at each of aplurality of output steps, a first higher order feature representationfor a corresponding acoustic frame in the sequence of acoustic frames; asecond encoder configured to: receive, as input, the first higher orderfeature representation generated by the first encoder at each of theplurality of output steps; and generate, at each of the plurality ofoutput steps, a second higher order feature representation for acorresponding first higher order feature frame; and a decoder configuredto: receive, as input, the second higher order feature representationgenerated by the second encoder at each of the plurality of outputsteps; and generate, at each of the plurality of time steps, a firstprobability distribution over possible speech recognition hypotheses. 9.The computer-implemented method of claim 8, wherein the decoder isfurther configured to: receive, as input, the first higher order featurerepresentation generated by the first encoder at each of the pluralityof output steps; and generate, at each of the plurality of time steps, asecond probability distribution over possible speech recognitionhypothesis.
 10. The computer-implemented method of claim 9, wherein thedecoder comprises: a prediction network configured to, at each of theplurality of time steps: receive, as input, a sequence of N previousnon-blank symbols output by a final Softmax layer; for each non-blanksymbol of the sequence of N previous non-blank symbols, generate arespective embedding; and generate an average embedding by averaging therespective embeddings; and a joint network configured to: receive, asinput, the average embedding generated by the prediction network at eachof the plurality of output steps and one of: when the ASR model isoperating in a streaming mode, the first higher order featurerepresentation generated by the first encoder at each of the pluralityof output steps; or when the ASR model is operating in a non-streamingmode, the second higher order feature representation generated by thesecond encoder at each of the plurality of output steps; and generate,at each of the plurality of output steps, one of: when the ASR model isoperating in the streaming mode, the second probability distributionover possible speech recognition hypothesis; or when the ASR model isoperating in the non-streaming mode, the first probability distributionover possible speech recognition hypothesis.
 11. Thecomputer-implemented method of claim 8, wherein: the first encodercomprises a causal encoder comprising an initial stack of conformerlayers; and the second encoder comprises a non-causal encoder comprisinga final stack of conformer layers overlain on the initial stack ofconformer layers.
 12. The computer-implemented method of claim 8,wherein the first encoder and the second encoder of the ASR model aretrained using Hybrid Autoregressive Transducer Factorization tofacilitate the integration of the external language model trained ontext-only data comprising the transcriptions from the set of trainingutterances and the identified subset of rare-word training text samples.13. A system comprising data processing hardware; and memory hardwarestoring instructions that when executed on the data processing hardwarecauses the data processing hardware to perform operations comprising:obtaining a set of training text samples; obtaining a set of trainingutterances used for training an automatic speech recognition (ASR)model, each training utterance in the plurality of training utterancescomprising audio data corresponding to an utterance and a correspondingtranscription of the utterance; applying rare word filtering on the setof training text samples to identify a subset of rare-word training textsamples that include words that do not appear in the transcriptions fromthe set of training utterances or appear in the transcriptions from theset of training utterances less than a threshold number of times; andtraining the external language model on the transcriptions from the setof training utterances and the identified subset of rare-word trainingtext samples.
 14. The system of claim 13, wherein obtaining the set oftraining text samples comprises: receiving a corpus of training textsamples; executing a resampling function on the corpus of training textsamples to identify high frequency text samples that occur in the corpusof training text samples; and obtaining the set of training text samplesby removing the identified high frequency text samples from the corpusof training text samples.
 15. The system of claim 13, wherein theresampling function comprises one of a simple power resampling function,a forced power resampling function, or a soft logarithmic resamplingfunction.
 16. The system of claim 13, wherein the operations furthercomprise: applying contrastive filtering on the set of training textsamples to identify a subset of target domain training text samples thatmatch a target domain associated with the set of training utterances,wherein training the external language model on the transcriptions fromthe set of training utterances and the identified subset of rare-wordtraining text samples further comprises training the external languagemodel on the identified subset of target domain training text samplesthat match the target domain.
 17. The system of claim 13, wherein theexternal language model comprises an external neural language model. 18.The system of claim 17, wherein the external neural language modelcomprises a stack of conformer layers or transformer layers.
 19. Thesystem of claim 13, wherein the operations further comprise integratingthe trained external language model with the trained ASR model, thetrained external language model configured to rescore probabilitydistributions over possible speech recognition hypotheses predicted bythe trained ASR model.
 20. The system of claim 19, wherein the ASR modelcomprises: a first encoder configured to: receive, as input, a sequenceof acoustic frames; and generate, at each of a plurality of outputsteps, a first higher order feature representation for a correspondingacoustic frame in the sequence of acoustic frames; a second encoderconfigured to: receive, as input, the first higher order featurerepresentation generated by the first encoder at each of the pluralityof output steps; and generate, at each of the plurality of output steps,a second higher order feature representation for a corresponding firsthigher order feature frame; and a decoder configured to: receive, asinput, the second higher order feature representation generated by thesecond encoder at each of the plurality of output steps; and generate,at each of the plurality of time steps, a first probability distributionover possible speech recognition hypotheses.
 21. The system of claim 20,wherein the decoder is further configured to: receive, as input, thefirst higher order feature representation generated by the first encoderat each of the plurality of output steps; and generate, at each of theplurality of time steps, a second probability distribution over possiblespeech recognition hypothesis.
 22. The system of claim 21, wherein thedecoder comprises: a prediction network configured to, at each of theplurality of time steps: receive, as input, a sequence of N previousnon-blank symbols output by a final Softmax layer; for each non-blanksymbol of the sequence of N previous non-blank symbols, generate arespective embedding; and generate an average embedding by averaging therespective embeddings; and a joint network configured to: receive, asinput, the average embedding generated by the prediction network at eachof the plurality of output steps and one of: when the ASR model isoperating in a streaming mode, the first higher order featurerepresentation generated by the first encoder at each of the pluralityof output steps; or when the ASR model is operating in a non-streamingmode, the second higher order feature representation generated by thesecond encoder at each of the plurality of output steps; and generate,at each of the plurality of output steps, one of: when the ASR model isoperating in the streaming mode, the second probability distributionover possible speech recognition hypothesis; or when the ASR model isoperating in the non-streaming mode, the first probability distributionover possible speech recognition hypothesis.
 23. The system of claim 20,wherein: the first encoder comprises a causal encoder comprising aninitial stack of conformer layers; and the second encoder comprises anon-causal encoder comprising a final stack of conformer layers overlainon the initial stack of conformer layers.
 24. The system of claim 20,wherein the first encoder and the second encoder of the ASR model aretrained using Hybrid Autoregressive Transducer Factorization tofacilitate the integration of the external language model trained ontext-only data comprising the transcriptions from the set of trainingutterances and the identified subset of rare-word training text samples.